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Abstract 

We show that the unified method recently proposed by Feldman 
and Cousins to put confidence intervals on bounded parameters can- 
not avoid the possibility of getting null results. A modified bayesian 
approach is also proposed (although not advocated) which ensures no 
null results and proper coverage. 



1 Introduction 

In a recent paper |IJ , Feldman and Cousins have revisited the long-standing 
problem of confidence belts on bounded parameters, for which the standard 
method proposed by Neyman leads in some cases to null, or unphysical, 
results, and have proposed a new method. 

The advocated method takes advantage of the freedom left by the Neyman 
construction in the choice of ordering used to select the ensemble of values 
of the measurement with a given probability content. 

This new method is strictly classical -or frequentist- (that is, not us- 
ing bayesian extensions to classical statistics), avoids overcoverage, gives a 
natural and non-biasing transition between upper limits and intervals, and, 
according to the authors, avoids null results. 
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We show in this paper that the last point is not strictly met and that null 
results can only be avoided for confidence levels above a limit which depends 
upon the probability law under consideration. 



2 The "problem" and its solutions 
2.1 Building confidence domains 

Let us consider a random variable x of density probability f(x\fj) where /x 
is an unknown parameter. Given an observation x, one wishes to make a 
statement on /x with a given confidence level (noted CL in the following) a. 
a is the probability that the statement is true. 

Let us first consider the case of an upper limit on /x. The Neyman con- 
struction consists in defining for each value of ji the value x m such that 

/Xm 
f(x\fi) dx = 1 - a (1) 
-co 

Whatever /x, x has a probability a to be bigger than x m (fi). x being observed, 
the a CL limit \im on /x is obtained by solving x m (/iM) = x. In the following, 
we will consider only the cases where this equation has a unique solution, 
implying that x m (/i) is a monotonic increasing function of \i for any value a. 
This is equivalent to stating that 

dF(x\u) , , 

^ <0Vx,|i (2) 

One can equally define an a CL interval [/x m , /xm] on /x by constructing for 
each ji an interval [x m (/x), Xm{h)] in x of probability content a and, given 
the observation x, solve x m (/iM) = XM(Hm) = x. But contrary to upper 
(or lower) limits, the choice of interval in x is not unique and an ordering 
prescription on x is necessary. 

The usual prescription consists in defining x m and x M by 

f(x\n) dx = / f(x\fi) dx=(l- a) /2 (3) 

-co Jxm 



(the so-called central interval). This only works for 1-dimensional x, and 
a new ordering for n-dimensional x has to be chosen. One generally uses 
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the x 2 between x and the mean value of x, x(fi), or the likelihood ratio 
R = f(x\n)/f(xo\(j,) where f(xo\[J>) is the maximum of /, and one defines a 
cut c on x 2 (resp R) so that the probability of x 2 < c (resp R < c) is a. 
These two methods (among others), when applied to the 1-dimensional case, 
would generally give non central intervals. 

2.2 The case of bounded parameters 

The method outlined above can lead to null results on the parameter if 
this parameter is bounded ( for example, fi > if fi is a mass, a variance, 
etc.). Such a case is illustrated on figure 1, where one sees that for low 
enough values of x, no upper limit on /x is obtained (or a negative limit is 
obtained in case f(x\fi) is defined even for negative /i's). 

For a statistician, this is not a problem, since an a CL statement has a 
(1 — a) probability to be wrong. For unbounded parameters, it is impossible 
to know if a statement is true or false. For bounded parameters, a fraction 
of the physicists emitting a wrong statement are aware that their statement 
is wrong, and would rather like not to be in this uncomfortable situation. 
One should stress however that their result, which when expressed as a limit 
on \i seems to bring no information, is as legitimate and useful as any other 
result obtained by those physicists publishing a physical limit (which might 
be right or wrong). 

But the discomfort caused by such a situation is so strong that cures have 
been searched for to avoid publishing null results. 

2.3 The bayesian solution 

The bayesian extension to classical statistics consists in building a probability 
density on the unknown parameter fi from the observation x by applying the 
(classical) Bayes theorem on conditional probabilities, af if [i were a random 
variable || . This gives a density probability (a degree of belief in bayesian 
language) 




(4) 



where K is a normalization coefficient insuring that 
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Figure 1: As a fuction of the parameter //, we show the acceptance domain 
in x obtained by the frequentist approach (above curve A defined by equa- 
tion 1) and by the bayesian approach (above curve B defined by equation 5). 
With curve A, null results are obtained for x < xo, while curve B gives 
over cover age. 
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and P(fi) summarizes our knowledge on fi prior to the observation x. In 
particular, for bounded parameters, P((jl) will be null outside the physical 
region. 

An upper limit at "a CL" /i can then be built on /j, by solving 



By construction, the selected range of physical fi values will never be 
empty, so that null results are impossible. This is shown on figure 1. 

This approach, although recommended by PDG [|J till 1997, has been 
criticized by frequentist statisticians for the following reasons: 

1. Near the parameter boundary, where null results happen with the stan- 
dard method, this bayesian " a CL" limit has a classical CL higher than 
a (it is actually 1 if \i happens to be below /ii, see figure 1), leading 
to overcoverage and loss of predictive power. Note however that this 
overcoverage is a local property, and could well transform into under- 
coverage at higher /i values. 

2. There is some arbitrariness in the choice of P(fi)- PDG proposes to 
use a Heavyside function to describe the physical limit on /i. Such a 
prescription has some drawbacks: it is not invariant under changes of 
parametrization, so that limits will actually depend on the parametriza- 
tion used. Furthermore, there are cases where the integral in (|^) is 
divergent unless -P(/i) is adequately chosen. 

I would like , in the next section, to propose a cure to the second point. 
2.4 A modified bayesian limit 

A classical upper limit on [i set at /xo from an observation x has a confidence 
level a equal to 1 — F(x\fio). If one is willing to interpret P{ji < /io) = a as a 
probability statement on /i, then the cumulative probability distribution for 
fi has to be 1 — F(x\fi), so that the probability density for \i deduced from 
the observation x is given by: 




(5) 




(6) 
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Contrary to the usual bayesian definition, g is always defined and normalized 
to 1 by construction. Furthermore, when equation |2] is satisfied, which we 
have supposed, g is positive. When \i is bounded, the definition of conditional 
probability can be applied to g to get its restriction to physical values. If 
fi > a, then (dropping x in the notation): 



will be the probability distribution of fi restricted to its physical values, g 
can then be used instead of g (defined in equation f|) to set an upper limit 
/io on /i, by solving: 



Using g rather than g has several advantages: 

• When ft is not bounded, the obtained limit is identical to the classical 
limit (for bounded parameters, it can be proven it gives overcoverage 
whatever the value of fi is, contrary to the usual bayesian method). 

• the limit is invariant under reparametrization. If we change fi to X — 
r(/x), the limit A will be r(/i ) 

• When prior knowledge is limited to the physical boundary, there is no 
need to invoke the ambiguous function P(fi), which is replaced by a 
Heavyside function irrespective of the parametrization. 

Let us note before closing this section that g(fi\x) = f(x\fi) in some special 
cases. Among them are the normal law of mean ft and constant variance, 
and the Poisson law of mean fi, so that for these cases, the limit given by g 
becomes identical to the former PDG recommendation. 

2.5 Feldman and Cousins solution 

Feldman and Cousins wanted to fulfill 3 conditions: 

1. Avoid null results 

2. keep a frequentist approach and produce results with a classical CL 
equal to a with no overcoverage (except when x is discrete, since this 
discreteness implies unavoidedly some overcoverage) 




(7) 




(8) 
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3. Solve the "flip-flop" problem of how to switch from upper limit to 
confidence interval, as they have shown that a choice made according 
to the value of the observation x leads, as is usually the case, to a biased 
result, namely an undercoverage (the actual CL being lower than the 
claimed one). This problem occurs in practice for parameters bounded 
from below, \x > a, where \x — a is the strength of an hypothetic signal 
on which to make a statement, and observations also bounded from 
below (x > xo). 

These authors propose an ordering principle on x based on 

() = f(x\») 

{) f(x\pi best ) (9j 

where Hbest is the physical value of /i which maximizes the denominator. r(x) 
varies between and 1, and one selects for each \x the values of x such that 
r(x) > r c and 

/ f(x\n)dx = a 

J r>r c 

This construction, being classical, meets the second condition. It will meet 
the first and third conditions in cases where the algorithm selects x values 
between x min {ji) and x max {n) such that x min = x for fi < a + s , and 
Xmin > Xo for /i > a + s , so that an upper limit is published when the 
observed value of x is below x max (a), and an interval on \x (not necessarily 
central) for higher values of x. 

If it is not the case, the algorithm will lead to null results: 

• If Xo is selected for all values of /x, the algorithm will produce lower 
limits for x > x max (a) and "null" results below (all physical values of 
/i being accepted). 

• If x is excluded for all values of fi, it will give a null result for 

x < x min (a), an interval for x > x max (ci) and an upper limit inbetween. 

We show in the next section that such a situation is likely to occur. 
2.6 Study of the ordering algorithm 

In order to prove the above mentionned fact, it is sufficient to exhibit cases 
where null results can be obtained. 
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I will restrict in the following to cases where [l and x both take only 
positive values, and where f{x\fj,) is , up to a normalization factor, a function 
g(y) with y — xj\i. If / g(y)dy = 1, the normalization factor is so that 
f(x\n) = g(y)/fi. (Such cases can be met in real cases: y can follow a x 2 
law and /i be the unknown variance, x then being the unnormalized x 2 from 
which one wishes to give a statement on the variance). Note that the limits 
on x at any confidence level are just straight lines passing thru the origin, 
and no problem of null results is encountered with the usual classical method. 
The problems appear only when for some reason, one restricts fi to values 
higher than a, a strictly positive constant (in the x 2 example, one knows 
that the variance is at least a, and one wishes to know if there is some other 
contribution to it). Note also that such distributions satisfy the condition of 
equation |2|. 

When no bound is imposed on /i (other than being positive), ^ e3t as 
defined in (|9]) is fi given by 



This equation is equivalent to 

^M = o 

where h(y) = yg(y) and y = x/fi . 

Let us suppose that h(y) is such that it has a unique maximum at y = y . 
To build r(x) when /i > a, two cases have to be considered: 

• when x > ay , fi > a, ^ best = /i and r(x) = h(y)/h(y ) 

• when x < ay , /i < a, fibest = a and r(x) = h(y) / h(fiy / a) 
Note that r(x = /i?/o) — 1; and 



where n is the first non-null derivative of g{y) at y — 0. 

When fi — > a, r(x) — > 1 for x between and ayo, then decreases. 







f(x\(jto) = 



dfx 
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Figure 2: r is shown as a function of y. h(y) takes its maximum at yo- The 
function r is made of 2 curves, curve A given by h(y)/h(yo) when x > ayo, 
curve B given by h(y) / h(fiy / a) when x < ayo. The curve B' corresponds to 
a value // of /i closer to a, and curve C is obtained when /i = a. 
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r(x) is shown on figure 2. We will suppose in the following that r(x) is a 
non decreasing function of x for x between and ayo, as it is the case in the 
explicit examples below (this is to avoid disconnected acceptance domains). 

Then, the acceptance domain for x contains x = /iy and extends to 
both sides according to the chosen value of a, including or not x — 0. When 
li — > a, the acceptance domain will or not contain x = depending on the 
value of / = Jq° g(y)dy. I is < 1. If / is smaller than a, the acceptance 
domain at fj, — a will include x = and no null results are possible. If / is 
bigger than a, the acceptance domain at /i — a will NOT include and null 
results will occur. 

One thus sees that the occurence or not of null results with Feldman and 
Cousins method will depend on the chosen CL. Contrary to the bayesian 
approach, the absence of null results is not insured by construction. 

2.7 Some examples 
2.7.1 exponential law 

g(y) = e~ v 

h(y) = ye~ y has a unique maximum at y — 1. 

I = Io9(y)dy =l-e- 1 = 0.63 

A CL higher than 0.63 will avoid null results. 

This law is a xl 1 &W (2 degrees of freedom) for 2y. More generally, x% 
laws give values of / slowly decreasing to 1 — 3e -2 = 0.594 when iV — > oo. 

Other usual laws tend to give values of / smaller than the usual choices 
of CL, 0.9 or higher, so that in practical cases, null results are likely to 
be avoided. It is however possible, by a careful choice of g(y), to obtain 
situations where / can be made as close to 1 as one wishes. For example, the 
sigmoid-like function: 

9 ^ = e (y-b) + i 

with y and b positive gives for large values of b an integral / whose value 
approaches 1 — log(6)/6 so that it can be made higher than any given CL. 

The next example, purely academic, exhibits a case where null results are 
unavoidable. 
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2.7.2 flat distribution 

g(y) = 1 for y between and 1 

x in this case is flat between and an unknown value /i restricted to be 
bigger than a. r(x) in this case is equal to a//i for x < a, and equal to xj\i 
for x between a and /i. The acceptance domain for x goes from (1 — to 
H (by connexity of the interval, higher values of x being selected first). 

Thus, whatever the chosen value for a, x = will be excluded from the 
acceptance domain for all values of /z, only lower limits will be obtained and 
null results will always occur. 

3 Discussion 

We have shown in this paper that the newly proposed method to put con- 
fidence belts on bounded parameters fails to meet some of its advocated 
properties, at least in principle although not in practice for most cases. 

Why should such a method be preferred to others? (Note that PDG, in 
its 1998 edition ||, recommends this new method). Feldman and Cousins 
argue that their method disantangles estimation and hypothesis testing, but 
one can argue as well that they obtain estimations from an ordering based 
on a quantity used for hypothesis testing! In view of the failure to avoid 
null results, I don't think any real argument can be given if one wishes to 
stick to classical statistics. In my opinion, different methods, as long as they 
give actual confidence levels equal to the announced one, are mathematically 
equally acceptable, and a comparison between them implies, consciously or 
not, some dose of "bayesian" input. 

The main problem at hand was to avoid null results. To cure this prob- 
lem (completely for the bayesian solution, partly only with Feldman and 
Cousins), one is led to break the symetry of central intervals which equally 
separates the wrong statements between the victims of fluctuations towards 
high values of x and victims of fluctuations towards low values. To avoid null 
results, a dissymetry is introduced to favor the latter to the detriment of the 
former, who anyway will never know if they are right or wrong. One could 
argue it is not fair! 

I would like to add that the good points of Feldman and Cousins' method 
could be translated to the bayesian approach to get rid of all the criticisms 
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Figure 3: The classical belt at a CL is given by curves Al and A2 defined 
by eqution 3. Bl is the bayesian upper limit obtained from formula 8 for a 
CL equal to (1 + a)/2. It is complemented by the curve B2 so that for each 
fi value, the probability content for x between Bl and B2 is a. 

it has received. We have already shown in section 2.4 how to solve the 
criticisms of arbitrariness by using a modified probability density for //. The 
only critics left is that of overcoverage for bayesian upper limits. But what 
Feldman and Cousins have shown is that one should publish intervals, which 
happen to be upper limits when the interval starts at the physical boundary. 
Thus, it would be perfectly legitimate, in order to avoid overcoverage, to 
build the "(1 + «)/2" upper limit with our modified bayesian recipee; it 
will show overcoverage, but this can be removed by implementing a lower 
limit on /i obtained by defining x max (fi) so that the probability content of 
selected x be a for any //. Such a construction would be devoid of any 
criticism : no overcoverage, no null results, no arbitrariness, and it concides 
with the classical central intervals of the Neyman construction for unbounded 
parameters. This is illustrated on figure 3. 
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The results obtained for Poisson statistics (in the case of a signal over 
a known background) would be the same as the usual (1 + a)/2 (instead of 
a, but this is the price to be paid to avoid flip-flop biasing) bayesian upper 
limit suitably complemented by a lower limit at high values of x, both curves 
conciding with the Neyman usual central interval for zero background. 

This method I think would certainly satisfy the advocates of the bayesian 
approach, and be acceptable by classical statisticians. Its construction might 
look kind of odd, but it uses as well as others the freedom left in the Neyman 
construction while fully responding to the anxiety created (with no good 
reason in my opinion) by the occurence of null results. 

I will close this paper with a final remark. 

All the recent discussions on the choice of method to be used, linked to 
the fact that different conclusions could be reached (for example, whether 
or not the recent Karmen result |3j excludes the LSND result |7J on neu- 
trino oscillations ||) have taken such an importance due to the tendancy to 
publish results at lower and lower confidence levels. Let me remind that the 
probability that 2 results being published at 90% CL are both right is only 
0.81 ! And with 90 % CL, a non negligible fraction of experiments can obtain 
null results. It would be in my opinion much more reasonable to publish 
results with CL of at least 99%, so that the intensity of discussions would 
considerably drop, because on the one hand, the fraction of null results would 
reach a very low level and on the other hand the probability of two results 
being both right would reach 98 %, whatever the algorithms used. Certainly 
results would look less spectacular, but would gain in fiability. 
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